feat(eval): grouped help and command examples for the eval commands by jariy17 · Pull Request #2262 · aws/agentcore-cli

jariy17 · 2026-09-09T21:55:30Z

I have decided to implement help groups using the Commander helpGroup feature. As shown below, organizing flags into groups makes it easier for users to understand how each flag functions within the command.

`agentcore eval batch-evaluation evaluate`

Before (refactor):

Usage: agentcore eval batch-evaluation evaluate [options]

evaluate existing sessions service-side (async; returns a job ID)

Options:
  --agent <agent>                            source: harness ID or Runtime ID whose sessions to use
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --online-eval <online-eval>                source: use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  source: raw DataSourceConfig JSON (inline, file://<path>, or -); escape hatch
  --start-time <start-time>                  time filter: window start (ISO-8601, with --end-time)
  --end-time <end-time>                      time filter: window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             filter: specific session IDs (only with --agent)
  --evaluators <evaluators...>               evaluator ID(s) to apply
  --ground-truth <ground-truth>              session ground truth (JSON SessionMetadataShape[]; inline, file://<path>, or -)
  --name <name>                              batch evaluation name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt evaluation data at rest
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

After (this PR):

Usage: agentcore eval batch-evaluation evaluate [options]

evaluate existing sessions service-side (async; returns a job ID)

Configuration:
  --name <name>                              batch evaluation name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt evaluation data at rest

Session source (choose exactly one):
  --agent <agent>                            harness ID or Runtime ID whose sessions to use
  --online-eval <online-eval>                use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  the traces to read (JSON DataSourceConfig); escape hatch

Source filters:
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --start-time <start-time>                  window start (ISO-8601, with --end-time)
  --end-time <end-time>                      window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             specific session IDs (only with --agent)

Evaluation:
  --evaluators <evaluators...>               evaluator ID(s) to apply
  --ground-truth <ground-truth>              expected answers for the sessions (JSON SessionMetadataShape[])

Other options:
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Where sessions and traces are read from, for sources the --agent and
      --online-eval convenience flags cannot express. Only top-level key:
      cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "logGroupNames": ["string", ...],  // [required] groups holding the traces
            "serviceNames": ["string", ...],   // e.g. "my_agent.DEFAULT"
            "filterConfig": {
              "sessionIds": ["string", ...],
              "sessionFilterConfig": {
                "startTime": "timestamp",
                "endTime": "timestamp"
              }
            }
          }
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/support_agent-AbC123XyZ9-DEFAULT"],"serviceNames":["support_agent.DEFAULT"],"filterConfig":{"sessionIds":["session-123"]}}}'

  --ground-truth (JSON: list of objects)
      Expected answers for the sessions being evaluated, so an evaluator can score a
      response against a reference instead of judging it on its own. Each entry names
      one session; omit an entry for a session that has no reference answer.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "sessionId": "string",         // [required] the session the reference applies to
            "testScenarioId": "string",    // groups sessions replaying the same scenario
            "groundTruth": {
              "inline": "string"           // the expected answer
            }
          },
          ...
        ]

      Example:
        --ground-truth '[{"sessionId":"session-123","groundTruth":{"inline":"The order shipped on Tuesday."}}]'

        --ground-truth file://ground-truth.json

`agentcore eval batch-evaluation simulate`

Before (refactor):

Usage: agentcore eval batch-evaluation simulate [options]

replay a dataset against a Runtime, then batch-evaluate the resulting sessions

Options:
  --runtime-id <runtime-id>                Runtime ID to invoke per scenario
  --qualifier <qualifier>                  Runtime endpoint qualifier (default DEFAULT)
  --payload-template <payload-template>    JSON payload template; {input} is the scenario input, e.g. {"prompt":"{input}"}
  --header <header...>                     an ordered application header (repeatable)
  --bearer-token <bearer-token>            CUSTOM_JWT bearer token (for JWT-auth Runtimes)
  --user-id <user-id>                      Runtime user ID
  --dataset <dataset>                      dataset source: local JSONL path or a dataset ID
  --dataset-version <dataset-version>      dataset version (with a dataset ID)
  --evaluators <evaluators...>             evaluator ID(s) to apply
  --name <name>                            batch evaluation name (unique in the account)
  --description <description>              description for the batch evaluation
  --kms-key-arn <kms-key-arn>              KMS key to encrypt evaluation data at rest
  --ingestion-wait-ms <ingestion-wait-ms>  ms to wait for span ingestion before grading (default 180000; 0 to skip)
  -h, --help                               display help for command

Global Options:
  --region <region>                        AWS region
  --debug                                  debug logging (default: false)
  --json                                   JSON output (default: false)
  --endpoint-url <endpoint-url>            endpoint URL override

After (this PR):

Usage: agentcore eval batch-evaluation simulate [options]

replay a dataset against a Runtime, then batch-evaluate the resulting sessions

Runtime invocation:
  --runtime-id <runtime-id>                Runtime ID to invoke per scenario
  --qualifier <qualifier>                  Runtime endpoint qualifier (default DEFAULT)
  --payload-template <payload-template>    request body per example (JSON object); {input} is replaced with the input
  --header <header...>                     an ordered application header (repeatable)
  --bearer-token <bearer-token>            CUSTOM_JWT bearer token (for JWT-auth Runtimes)
  --user-id <user-id>                      Runtime user ID

Dataset:
  --dataset <dataset>                      dataset source: local JSONL path or a dataset ID
  --dataset-version <dataset-version>      dataset version (with a dataset ID)
  --ingestion-wait-ms <ingestion-wait-ms>  ms to wait for span ingestion before grading (default 180000; 0 to skip)

Configuration:
  --name <name>                            batch evaluation name (unique in the account)
  --description <description>              description for the batch evaluation
  --kms-key-arn <kms-key-arn>              KMS key to encrypt evaluation data at rest

Evaluation:
  --evaluators <evaluators...>             evaluator ID(s) to apply

Other options:
  -h, --help                               display help for command

Global Options:
  --region <region>                        AWS region
  --debug                                  debug logging (default: false)
  --json                                   JSON output (default: false)
  --endpoint-url <endpoint-url>            endpoint URL override

Parameter details:

  --payload-template (JSON object)
      The request body sent to the Runtime for each dataset example. Every occurrence
      of {input} is replaced with that example's input, so the template describes the
      shape your agent expects and {input} marks where the prompt goes.

      Example:
        --payload-template '{"prompt":"{input}"}'

        --payload-template '{"messages":[{"role":"user","content":"{input}"}],"stream":false}'

`agentcore eval batch-insights run`

Before (refactor):

Usage: agentcore eval batch-insights run [options]

start an asynchronous batch insights run over existing sessions

Options:
  --agent <agent>                            source: harness ID or Runtime ID whose sessions to use
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --online-eval <online-eval>                source: use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  source: raw DataSourceConfig JSON (inline, file://<path>, or -); escape hatch
  --start-time <start-time>                  time filter: window start (ISO-8601, with --end-time)
  --end-time <end-time>                      time filter: window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             filter: specific session IDs (only with --agent)
  --insight <insight...>                     insight ID(s) to run (default: ["Builtin.Insight.FailureAnalysis"])
  --evaluators <evaluators...>               optional evaluator ID(s) to run alongside the insights
  --name <name>                              batch insights name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt insights data at rest
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

After (this PR):

Usage: agentcore eval batch-insights run [options]

start an asynchronous batch insights run over existing sessions

Configuration:
  --name <name>                              batch insights name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt insights data at rest

Session source (choose exactly one):
  --agent <agent>                            harness ID or Runtime ID whose sessions to use
  --online-eval <online-eval>                use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  the traces to read (JSON DataSourceConfig); escape hatch

Source filters:
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --start-time <start-time>                  window start (ISO-8601, with --end-time)
  --end-time <end-time>                      window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             specific session IDs (only with --agent)

Analysis:
  --insight <insight...>                     insight ID(s) to run (default: ["Builtin.Insight.FailureAnalysis"])
  --evaluators <evaluators...>               optional evaluator ID(s) to run alongside the insights

Other options:
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Where sessions and traces are read from, for sources the --agent and
      --online-eval convenience flags cannot express. Only top-level key:
      cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "logGroupNames": ["string", ...],  // [required] groups holding the traces
            "serviceNames": ["string", ...],   // e.g. "my_agent.DEFAULT"
            "filterConfig": {
              "sessionIds": ["string", ...],
              "sessionFilterConfig": {
                "startTime": "timestamp",
                "endTime": "timestamp"
              }
            }
          }
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/support_agent-AbC123XyZ9-DEFAULT"],"serviceNames":["support_agent.DEFAULT"],"filterConfig":{"sessionIds":["session-123"]}}}'

`agentcore eval online-eval create`

Before (refactor):

Usage: agentcore eval online-eval create [options]

create an online evaluation config

Options:
  --name <name>                                        the name of the online evaluation config
  --agent <agent>                                      harness ID or Runtime ID whose traffic to sample
  --endpoint <endpoint>                                the agent endpoint qualifier to scope monitoring to (default DEFAULT)
  --data-source-config <data-source-config>            the traces to evaluate (JSON DataSourceConfig; inline, file://<path>, or - for stdin), as an alternative to --agent
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440, default 15)
  --filters <filters>                                  trace filters (JSON Filter[]; inline, file://<path>, or - for stdin)
  --role-arn <role-arn>                                IAM role the online evaluation assumes (default auto-provisioned)
  --enable-on-create <enable-on-create>                whether to enable evaluation immediately (default true; pass false to create it paused)
  --description <description>                          a description of the config's monitoring purpose
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

After (this PR):

Usage: agentcore eval online-eval create [options]

create an online evaluation config

Configuration:
  --name <name>                                        the name of the online evaluation config
  --description <description>                          a description of the config's monitoring purpose
  --enable-on-create <enable-on-create>                whether to enable evaluation immediately (default true; pass false to create it paused)

Session source (choose exactly one):
  --agent <agent>                                      harness ID or Runtime ID whose traffic to sample
  --data-source-config <data-source-config>            the traces to sample (JSON DataSourceConfig), as an alternative to --agent
  --endpoint <endpoint>                                the agent endpoint qualifier to scope monitoring to (default DEFAULT)

Evaluation:
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440, default 15)
  --filters <filters>                                  trace filters (JSON Filter[])

Execution:
  --role-arn <role-arn>                                IAM role the online evaluation assumes (default auto-provisioned)

Other options:
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Which traces are sampled, for sources the --agent convenience flag cannot
      express. Only top-level key: cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "serviceNames": ["string", ...],          // [required] e.g. "my_agent.DEFAULT"
            "logGroupNames": ["string", ...],         // exact group names
            "logGroupNamePrefixes": ["string", ...]   // or match by prefix instead
          }                                           // supply one of the two name lists
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore-control/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/my-runtime-DEFAULT"],"serviceNames":["my_agent.DEFAULT"]}}'

  --filters (JSON: list of objects)
      Narrows which sampled traces are evaluated. A trace must match every filter in
      the list. Omit it to evaluate every sampled trace.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "key": "string",       // [required] trace field to filter on
            "operator": "Equals" | "NotEquals" | "Contains" | "NotContains"
                      | "GreaterThan" | "GreaterThanOrEqual"
                      | "LessThan" | "LessThanOrEqual",   // [required]
            "value": {                                    // [required] exactly one key
              "stringValue": "string",
              "doubleValue": number,
              "booleanValue": true | false
            }
          },
          ...
        ]

      Example:
        --filters '[{"key":"attributes.customer_tier","operator":"Equals","value":{"stringValue":"enterprise"}}]'

        --filters file://filters.json

`agentcore eval online-eval update`

Before (refactor):

Usage: agentcore eval online-eval update [options]

update an online evaluation config

Options:
  --id <id>                                            the ID of the online evaluation config to update
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440)
  --filters <filters>                                  trace filters (JSON Filter[]; inline, file://<path>, or - for stdin)
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply (replaces the existing list)
  --agent <agent>                                      repoint at a different harness ID or Runtime ID
  --endpoint <endpoint>                                re-scope monitoring to a different agent endpoint qualifier
  --clear-endpoint <clear-endpoint>                    reset the endpoint scope to the default qualifier (pass true)
  --data-source-config <data-source-config>            replace the traces to evaluate (JSON DataSourceConfig; inline, file://<path>, or - for stdin)
  --role-arn <role-arn>                                replace the IAM role the online evaluation assumes
  --update-role <update-role>                          whether to re-scope an auto-provisioned execution role when the data source changes (default true)
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

After (this PR):

Usage: agentcore eval online-eval update [options]

update an online evaluation config

Target:
  --id <id>                                            the ID of the online evaluation config to update

Session source:
  --agent <agent>                                      repoint at a different harness ID or Runtime ID
  --data-source-config <data-source-config>            replace the traces to sample (JSON DataSourceConfig)

Source filters:
  --endpoint <endpoint>                                re-scope monitoring to a different agent endpoint qualifier
  --clear-endpoint <clear-endpoint>                    reset the endpoint scope to the default qualifier (pass true)

Evaluation:
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply (replaces the existing list)
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440)
  --filters <filters>                                  replace the trace filters (JSON Filter[])

Execution:
  --role-arn <role-arn>                                replace the IAM role the online evaluation assumes
  --update-role <update-role>                          whether to re-scope an auto-provisioned execution role when the data source changes (default true)

Other options:
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Which traces are sampled, for sources the --agent convenience flag cannot
      express. Only top-level key: cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "serviceNames": ["string", ...],          // [required] e.g. "my_agent.DEFAULT"
            "logGroupNames": ["string", ...],         // exact group names
            "logGroupNamePrefixes": ["string", ...]   // or match by prefix instead
          }                                           // supply one of the two name lists
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore-control/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/my-runtime-DEFAULT"],"serviceNames":["my_agent.DEFAULT"]}}'

  --filters (JSON: list of objects)
      Narrows which sampled traces are evaluated. A trace must match every filter in
      the list. Omit it to evaluate every sampled trace.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "key": "string",       // [required] trace field to filter on
            "operator": "Equals" | "NotEquals" | "Contains" | "NotContains"
                      | "GreaterThan" | "GreaterThanOrEqual"
                      | "LessThan" | "LessThanOrEqual",   // [required]
            "value": {                                    // [required] exactly one key
              "stringValue": "string",
              "doubleValue": number,
              "booleanValue": true | false
            }
          },
          ...
        ]

      Example:
        --filters '[{"key":"attributes.customer_tier","operator":"Equals","value":{"stringValue":"enterprise"}}]'

        --filters file://filters.json

Tests

Router (6 new): heading names and ordering, ungrouped flags staying in Options:, -h placement with and without groups, examples rendering exactly once after Parameter details:, backslash continuation for array commands vs single-line for strings, and the no-examples case.

Handlers (14 new, src/handlers/eval/helpGroups.test.ts): asserts what the router's rendering tests cannot — that every flag on these five commands carries a group, that the group comes from the shared vocabulary, that a heading's flags are declared contiguously (a stray one would reorder the headings), and that no example names a flag its command doesn't declare.

Mutation-checked: ungrouping a flag, hand-typing a heading that drifts from the vocabulary, and misspelling a flag inside an example each fail a test.

Verification

bun test — 3199 pass, 0 fail (225 files)
bun run typecheck, bun run lint:check, bun run format:check — clean
No existing test needed changing: the *.screen.test.tsx suites don't assert on these commands' option layout

Follow-ups

#2265 adds --output-config to batch-evaluation, renames --qualifier → --endpoint, and makes --start-time/--end-time require a timezone. #2266 does the online-eval equivalents plus the managed-role widening.

agentcore-devx-automation · 2026-09-09T21:57:24Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Reviewed the router mechanism (flags.tsx, handler.tsx, router.tsx, index.tsx), the --evaluator → --evaluators rename across the 8 handlers, and the SDK bump.

The Flag.group plumbing is minimal and correctly optional; ungrouped flags stay in Commander's default section.
Handler.examples is optional and read off the authored node in compile(), so middleware wrappers (which only need to forward handle) don't have to carry it. withEffectiveTuiSupport also forwards it via handler.examples?.() so nothing gets dropped through TUI wrapping.
The -h, --help relocation to Other options: correctly triggers only when ownFlags.some(f => f.group), keeping unchanged behavior for commands that don't opt in.
New router tests cover: heading ordering, ungrouped flags remaining in Options:, -h placement in both cases, examples rendering exactly once after Parameter details:, and the no-examples case.
The rename is complete: no leftover "--evaluator" argv, flag("evaluator", …), or flags["evaluator"] reads in src/, and the singular evaluator resource-type enum in project remove is correctly left alone (it's a resource selector, not a flag list). Error messages updated to '--evaluators <evaluators...>'. All corresponding fixture/unit tests updated in the same PR.
SDK bump to 3.1129.0 on both agentcore clients is scoped and justified for the follow-up PRs.

No changes requested.

codecov-commenter · 2026-09-09T22:01:59Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.06%. Comparing base (551ab96) to head (e580808).
⚠️ Report is 3 commits behind head on refactor.

Additional details and impacted files

@@            Coverage Diff             @@
##           refactor    #2262    +/-   ##
==========================================
  Coverage     97.05%   97.06%            
==========================================
  Files           566      568     +2     
  Lines         39228    39340   +112     
==========================================
+ Hits          38073    38185   +112     
  Misses         1155     1155

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-09-09T22:22:11Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-09-10T22:05:16Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-09-10T22:59:07Z

Claude Security Review: no high-confidence findings. (run)

Two presentation mechanisms for commands whose option list has grown past the point of skimming. No command uses either yet — the follow-up PRs that add flags to batch-evaluation and online-evaluation consume them. Help groups: - Flag.group renders through Commander's Option.helpGroup(), so a command can carry semantic headings instead of one flat "Options:". Ungrouped flags keep Commander's default section, so nothing changes for commands that opt out. - A command that groups its flags gets its generated -h/--help moved to "Other options:". Left alone it sits by itself under "Options:", reading as though it were the command's only ungrouped flag. Examples: - Handler.examples takes { description, command } entries and the renderer owns the layout. `command` is authored as the shell command with no indentation: a string renders on one line, an array joins with backslash continuations. Splitting the array is how the author picks the break points, which keeps a terminal-width heuristic out of it and keeps the printed command pasteable. - commandExamples() mirrors commandParameterDetails(): Commander omits added help text from helpInformation(), so a TUI rendering has to ask for it or silently drop it.

Applies the router's group and example mechanisms to the five eval commands whose option lists had grown past skimming. Presentation only — no flag is added, removed, or renamed, and no request changes. One shared vocabulary (src/handlers/eval/helpGroups.tsx) rather than per-command headings: batch evaluation, batch insights, and online evaluation all name a session source, narrow it, and apply evaluators, so the same concept should read the same in all three. Naming the headings in one place also means a typo cannot silently split one heading into two. Notable placements: - `--name`/`--description`/`--kms-key-arn` sit under "Configuration:" rather than an "Evaluation options:" heading that would also have to hold encryption. - `--endpoint` joins "Session source:" on online-eval create instead of taking a heading of its own for one flag. - SessionSource declares its source arms before its filters, because Commander orders headings by the first flag declared in each. Its descriptions lose the "source:"/"filter:"/"time filter:" prefixes, which the headings now carry. Examples are worked invocations per command, covering each source arm and each authentication and dataset mode. batch-insights run gets groups but no examples. The new test asserts what the router's own rendering tests cannot: that every flag on these commands carries a group, that the group comes from the shared vocabulary, that a heading's flags are declared contiguously, and that no example names a flag its command does not declare.

…bulary Review feedback on #2262: - Each handler names its own `--help` headings; the shared HELP_GROUP module is gone, along with its unit test. - The Examples mechanism is removed entirely — Handler.examples, formatExamples, commandExamples, and the router tests that covered them. Worked invocations now live in each flag's `help:` block, which `--help` renders under "Parameter details" the way `harness create` already does. - Code comments added by the previous two commits are removed. Flags taking an API-shaped object gain that long-form documentation: --data-source-config (separately for the data-plane and control-plane shapes, which differ), --ground-truth, --filters, and --payload-template. Their one-line descriptions shrink to a type name, matching harness create.

Each handler declares the headings it uses once at the top of its own file rather than repeating the string at every flag — "Session source (choose exactly one):" was written three times in online-eval create alone. Declared in first-appearance order so reading them top-down matches the order the headings render. Nothing is shared between files: CONFIGURATION is declared independently in the four handlers that use it, and sessionSource owns SESSION_SOURCE and SOURCE_FILTERS because it declares those flags. `--help` output is byte-identical for all five commands.

agentcore-devx-automation · 2026-09-10T23:06:04Z

Claude Security Review: no high-confidence findings. (run)

A const earns its place when the string appears more than once; naming a single use only adds a hop. simulate's Evaluation, online-eval create's Execution, and update's Target headings are each used by one flag, so they go back to literals. The rest stay named.

agentcore-devx-automation · 2026-09-11T17:16:03Z

Claude Security Review: no high-confidence findings. (run)

notgitika

added comments for 4 bugs my agent found otherwise LGTM

notgitika · 2026-09-11T17:11:32Z

+    {
+      "sessionId": "string",         // [required] the session the reference applies to
+      "testScenarioId": "string",    // groups sessions replaying the same scenario
+      "groundTruth": {


this documented shape doesn’t match SessionMetadataShape. groundTruth.inline must be an InlineGroundTruth object, not a string. The current example will produce an invalid API request. Could we update it to use a valid shape such as {"inline":{"turns":[...]}}?

notgitika · 2026-09-11T17:14:06Z

+      "serviceNames": ["string", ...],   // e.g. "my_agent.DEFAULT"
+      "filterConfig": {
+        "sessionIds": ["string", ...],
+        "sessionFilterConfig": {


this property should be timeRange, not sessionFilterConfig. the SDK serializer drops the unknown property, so these dates are silently omitted and the evaluation may include sessions outside the intended window

notgitika · 2026-09-11T17:19:38Z

+  {
+    "cloudWatchLogs": {
+      "serviceNames": ["string", ...],          // [required] e.g. "my_agent.DEFAULT"
+      "logGroupNames": ["string", ...],         // exact group names


The locked control-plane SDK does not support logGroupNamePrefixes; CloudWatchLogsInputConfig requires both logGroupNames and serviceNames. Following the documented prefix alternative sends no logGroupNames and fails service validation.
Could we remove that alternative?

notgitika · 2026-09-11T17:20:27Z

This grouping is lost in the interactive TUI. These eval commands render through CliOnlyScreen, which flattens visibleOptions() into one “options” table without consulting each option’s help group. Could we preserve the groups there as well so the CLI and TUI help remain consistent?

github-actions Bot added the size/m PR size: M label Sep 9, 2026

agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Sep 9, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 9, 2026

agentcore-devx-automation Bot reviewed Sep 9, 2026

View reviewed changes

agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Sep 9, 2026

jariy17 force-pushed the feat/eval-cli-router-groups branch from 662bc9b to e6da03d Compare September 9, 2026 22:20

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Sep 9, 2026

jariy17 changed the title ~~feat(router): add help groups and command examples, rename --evaluator to --evaluators~~ feat(router): add help groups and command examples Sep 9, 2026

jariy17 changed the base branch from refactor to feat/rename-evaluators September 9, 2026 22:20

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Sep 9, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 9, 2026

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Sep 9, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 9, 2026

jariy17 had a problem deploying to e2e-testing September 9, 2026 23:17 — with GitHub Actions Failure

github-actions Bot added size/l PR size: L and removed size/m PR size: M labels Sep 9, 2026

jariy17 changed the title ~~feat(router): add help groups and command examples~~ feat(eval): grouped help and command examples for the eval commands Sep 9, 2026

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Sep 9, 2026

jariy17 mentioned this pull request Sep 9, 2026

feat(batch-evaluation): add --output-config, rename --qualifier to --endpoint, require timestamp timezones #2265

Draft

jariy17 added this pull request to stack #2270 September 10, 2026 15:43

jariy17 force-pushed the feat/eval-cli-router-groups branch from 560bd65 to b14c0dc Compare September 10, 2026 15:53

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 10, 2026

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Sep 10, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 10, 2026

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Sep 10, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 10, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 10, 2026

jariy17 added 4 commits September 10, 2026 23:00

jariy17 force-pushed the feat/eval-cli-router-groups branch from ca176e6 to c8936e9 Compare September 10, 2026 23:04

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Sep 10, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 10, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 10, 2026

jariy17 marked this pull request as ready for review September 10, 2026 23:12

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Sep 11, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 11, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 11, 2026

notgitika reviewed Sep 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(eval): grouped help and command examples for the eval commands - #2262

feat(eval): grouped help and command examples for the eval commands#2262
jariy17 wants to merge 5 commits into
refactorfrom
feat/eval-cli-router-groups

jariy17 commented Sep 9, 2026 •

edited

Loading

Uh oh!

agentcore-devx-automation Bot commented Sep 9, 2026

Uh oh!

agentcore-devx-automation Bot left a comment

Uh oh!

codecov-commenter commented Sep 9, 2026 •

edited

Loading

Uh oh!

agentcore-devx-automation Bot commented Sep 9, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 10, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 10, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 10, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 11, 2026

Uh oh!

notgitika left a comment

Uh oh!

notgitika Sep 11, 2026

Uh oh!

notgitika Sep 11, 2026

Uh oh!

notgitika Sep 11, 2026

Uh oh!

notgitika Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jariy17 commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

agentcore eval batch-evaluation evaluate

agentcore eval batch-evaluation simulate

agentcore eval batch-insights run

agentcore eval online-eval create

agentcore eval online-eval update

Tests

Verification

Follow-ups

Uh oh!

agentcore-devx-automation Bot commented Sep 9, 2026

Uh oh!

agentcore-devx-automation Bot left a comment

Choose a reason for hiding this comment

AgentCore Harness Review

Uh oh!

codecov-commenter commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

agentcore-devx-automation Bot commented Sep 9, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 10, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 10, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 10, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 11, 2026

Uh oh!

notgitika left a comment

Choose a reason for hiding this comment

Uh oh!

notgitika Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

notgitika Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

notgitika Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

notgitika Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jariy17 commented Sep 9, 2026 •

edited

Loading

`agentcore eval batch-evaluation evaluate`

`agentcore eval batch-evaluation simulate`

`agentcore eval batch-insights run`

`agentcore eval online-eval create`

`agentcore eval online-eval update`

codecov-commenter commented Sep 9, 2026 •

edited

Loading